[DSV4.1] Fuse DSpark verify compression, indexer and projections - #39068
Conversation
Fuse indexer score filtering and page mapping, candidate mask publication, Q RoPE buffer stores, and WO-A reduction with MXFP8 quantization. Co-authored-by: DarkSharpness <2040703891@qq.com>
Integrate the request-major verify kernel with fixed-width target batches. Retain compact-ragged and prefill fallback paths; validate ring wrap, accepted-prefix rollback, graph replay and real pool writes. Co-authored-by: DarkSharpness <ziyi.xu@radixark.ai>
BBuf
changed the title
…-project#39068) Co-authored-by: DarkSharpness <2040703891@qq.com> Co-authored-by: DarkSharpness <ziyi.xu@radixark.ai> (cherry picked from commit 0d5e663)
|
Additional bounded serving validation on four DGX Sparks / GB10, for the changes at We applied this patch set over SGLang The candidate passed all seven API keys, vision/tools, exact retrieval through 406713 prompt tokens, chunk/tail-boundary and mixed prefill/decode gates. All 20 first-token outputs and all 20 greedy continuations (up to 48 tokens) matched the same fixed reference as the preceding arm; four prompts are longer inputs. Minimum sampled host MemAvailable was 23.383 GiB. No decode request errors occurred. Performance is preliminary. Our three C8 aggregate-throughput samples, each with exactly 2048 completion tokens, were:
Although the medians are 90.33 and 111.15, these distributions overlap substantially. The harness uses new random/nonce prompts each wave, temperature 0, thinking off, forced 256 output tokens per request, and aggregate tokens divided by whole-wave wall time including prefill. Natural speculative acceptance is not held constant. These observations therefore do not establish a 23% gain or reproduce the PR's simulated-acceptance 12.15% result. We provisionally retained the patch under our local selection rule, but a fixed-input, warmed, repeated comparison is needed to attribute a throughput improvement. A subsequent reboot and full requalification of this same memory + #39068 profile passed, with a C8 median of 88.21 tok/s and a 23.156 GiB sampled memory floor. That repeat further cautions against treating the initial median increase as a reproducible gain. The useful confirmation so far is bounded GB10 serving compatibility and output agreement for this combination. SparkRing PR267 records the original deployment/reference; its completed soak predates these additional patches. |
|
Qualification to the bounded output-agreement result above: another reboot of the same memory + #39068, block-5 profile returned 20/20 matching first tokens but 19/20 exact continuations, versus 20/20 on the preceding reboot. The single differing continuation was a short regex fixture; launch receipts were byte-identical. Auth, vision/tools and long retrieval still passed. This does not identify #39068 as the cause, but means exact continuation agreement across boots has not been established. We stopped further tuning pending recovery and investigation. Details are in the control-repeatability follow-up. Further diagnostic evidence: the recovered block-5 profile then produced identical text in 80 short C1 requests (four fixed fixtures, ten repeats each, with and without continuation logprobs). At the historical regex divergence position, the new passing capture reports exactly equal logprobs for Controlled follow-up now complete on the same pins/configuration described above: one warm B observation, then fresh-boot A/B/B/A, where A is the memory backport alone and B adds this PR. Block5 and the remaining launch settings stayed fixed; full source manifests and selected configuration matched across all four ranks before/after every arm. Each arm captured288 measured requests (six fixed synthetic prompts, before/after identical conditioning history, fresh/reused prefix namespaces, ten logged plus two plain responses per condition). Padded prefixes positively exercised512 cached tokens; fresh namespaces reported0. Temperature0, up to48 output tokens, native speculative acceptance; thinking and seed omitted consistently. All1440 measurements carried native speculative statistics, and all240 arithmetic plus240 exact-copy controls were correct. A padded sky prompt (748 input tokens) produced7 distinct completions in warm B, then10/10/7/5 in fresh A/B/B/A, each among48 measurements. Variation appeared in both memory-only boots, before the conditioning workload, with0 and512 cached tokens, and with/without logprobs. Thus #39068 is not required for this symptom. These small distinct-output counts do not estimate a patch effect on its rate. The original short sky prompt also varied once in the second fresh B boot; regex and the objective controls remained consistent throughout this matrix. Some paired captures after identical generated prefixes changed candidate preference with nonzero displayed logprob gaps, so the padded-prompt symptom is broader than the exact regex tie previously observed. This does not identify a responsible kernel or establish semantic degradation. Both arms still share the memory backport, Mia/speculative execution and low-precision/distributed paths. We are treating this as a shared-stack repeatability investigation, not evidence that this PR introduces a correctness regression. No additional throughput claim is supported by these short, variable-output diagnostics. A further on/off/on control has now completed with the memory backport + this PR fixed throughout, using the same six-fixture protocol (288 measured requests per arm). The first on arm used the recovered warm server; off and return-to-on each used a fresh boot. All four ranks' actual launch flags and source/configuration receipts were verified before/after. Off omitted the speculative-algorithm flags and produced no speculative statistics or positive draft/verify counters. Distinct completions among48 observations per fixture:
Arithmetic, exact copy and padded regex each had one output in every arm; all144 arithmetic and144 copy controls were correct. Fresh prefixes reported0 cached tokens, reused padded prefixes512. Off-mode padded-sky variation appeared both before/after conditioning and with/without logprobs. An off-mode short-sky pair with identical generated prefix changed preference at token11: This rules out speculative execution as a requirement for the observed variation. It does not identify a shared operator, prove a semantic regression, or establish that speculation increases/decreases its frequency: these are small descriptive samples, and changing speculation also changes execution shapes and paths. Combined with the preceding memory-only control, we have not isolated this symptom to #39068 or to target verification. Further work should focus on the remaining shared execution paths; no additional performance claim follows from this diagnostic. Final bounded serving qualification on the same pins and memory + this PR combination: context 655360, requested KV 1500000 (actual 1499904), block5 and other tuning unchanged. Three-depth retrieval passed at approximately 647K input tokens; minimum sampled host MemAvailable was22.849GiB. A fresh boot passed112 structured-output requests across nonstreaming chat, streaming chat and raw generation: fixed tool-result continuations plus auto/forced tool calls, serial and C8. Checks required expected values/arguments, normal stopping and no leaked/repeated tool protocol. Raw results were inspected. These reconstructed fixtures are not an exact replay of an external corruption report and do not establish a universal fix or resolve the wording-variation observations above. The final20-minute C8 soak passed736 requests with zero failures/hangs;168 reached the configured output cap. Aggregate-throughput median106.35tok/s is workload-specific, not a patch speedup estimate. A separate1749760-token allocated KV pool passed eight distinct262144-token exact-answer/normal-stop pressure requests (2097152 total tokens) and recent-prefix reuse; this tests retained-cache eviction, not simultaneous full-pool live contexts. We retained1.5M under our performance selection rule. Its C8 samples varied from roughly86 to113tok/s across these boots, so the larger-pool nonpromotion does not establish a causal KV slowdown. No additional patch performance attribution follows from this qualification. |
2 participants
Summary
DSpark target verify launches many short kernels around compression, sparse-index selection and attention projections. On 4×GB300, TP4/EP4, BS1 random 4096-input / 1024-output tokens with simulated acceptance target 5.5, the PR improves median streamed decode throughput from the recorded 761.03 to 853.49 tokens/s (+12.15%). The C2 verify integration adds 6.37% over the preceding 802.38-token/s version.
Mxfp8SwizzledInputdistinguishes these inputs from ordinary block-FP8 tuples; the existing plain-tuple path is unchanged.The fast paths retain shape/backend guards. The tested head is
835c39094ad017c2f54f8ea598002e669e6fa30d, from a clean checkout; source and installed-package manifests are checked before and after each validation phase.Correctness
test_dsv4_indexer_postprocess.py: 3 tests passed, including exact reference checks, invalid/NaN/Inf values, ties, strided metadata, production candidate dimensions, underfilled rows and CUDA Graph replay.test_dsv4_q_rope_store.py: 2 tests passed, with bitwise comparison to existing CUDA RoPE + copy, unchanged input/padding and graph replay.test_block_fp8_as_mxfp8.py: 2 tests passed. The added test compares FP8 bytes, swizzled scales and full WO-B outputs exactly for M=2..8; covers strided WO-A input, graph replay with changed data/poisoned scale padding, and legacy tuple dispatch.Repository pre-commit hooks passed on all changed files.
C2 kernel and backend integration tests: 33 passed, 4 subtests passed, 4 all-padding combinations skipped. Cover request-major verify with 2/5/6/9 rows, padding, both metadata integer dtypes, ring wrap, every accepted-prefix length followed by rollback, CUDA Graph replay, reference arithmetic and real cache writes. Compact-ragged fallback dispatch is also checked.
Dataset accuracy
Acceptance simulation is disabled for all accuracy runs. DSpark uses genuine draft/target agreement, with the same checkpoint revision as the performance test. Per the requested protocol, the prior completed accuracy run is reused instead of rerunning a baseline.
Historical reference:
0669e3d9464con 4×B300. Current candidate:835c39094ad017c2f54f8ea598002e669e6fa30don 4×GB300. These are historical accuracy comparisons across different code revisions and hardware, not a fresh paired A/B of this PR's parent.GSM8K: 1314 held-out questions, with the first five test rows used only as demonstrations; temperature 0, top-p 1, seed 0, max output 4096, legacy prompt/scorer. Both current GSM lanes have zero empty answers. The concurrent lane's one truncated answer repeatedly debates the wording “two times more”; it remains an incorrect sample in the denominator.
AIME 2026:
sgl-eval==0.1.0, MathArena prompt, thinking enabled, reasoning effort max, top-p 0.95, max output 65536. Serial: 30 questions, temperature 0, seed 0, concurrency 1. Repeated: 16 responses per question (480 total), temperature 1, no fixed request seed, concurrency 64. Report correctness over all responses, not pass@16; retain truncations and errors. Historical AIME truncations were 3/30 and 15/480 respectively, with zero request errors.All four evaluation lanes completed. No lower aggregate score than the historical reference was observed in the matched serial/repeated lanes; this does not establish bitwise equivalence or isolate hardware effects.
Performance
Previously recorded PR baseline:
3b709e55c0f7599f90bdd400e1fe758c5a942cb6. Pre-C2 integration:2e4dff1c4939c8589191e65340236b03b54dc84c. Current candidate:835c39094ad017c2f54f8ea598002e669e6fa30d. Model revision:dba1be0a40aa45a94ad051997016db3960a90277.4×GB300, TP4/EP4, BS1, DSpark block size 5, static verify,
SGLANG_SIMULATE_ACC_LEN=5.5,match-expected. PyTorch 2.13.0+cu130, Triton 3.7.1, FlashInfer 0.6.18, sglang-kernel 0.4.6.post1, sgl-deep-gemm 0.1.7, CUTLASS DSL 4.6.2.Every request used identical input IDs and completed exactly 1024 output tokens. Exclude one warmup per launch and retain all subsequent rounds, including low outliers. Observed acceptance median is 5.520256 for the current PR, versus 5.505376 in the recorded earlier runs; the configured target remains 5.5. Throughput is
(final completion tokens - first streamed event tokens) / (last event time - first event time)and excludes prefill. Timing measurements run without the profiler. Simulation measures runtime performance, not natural model agreement or answer quality; it disables the graph-internal acceptance path.All current candidate measurements (tokens/s)
TP0 GPU traces cover 20 target/draft cycles. The pre-C2 trace has 40,673 kernels; the current trace has 37,313 (168 fewer per cycle). The current trace contains 60 fused C2 verify kernels: three per target pass, at L2/L8/L14. A separate CPU+GPU trace confirms 15 calls in five passes. Layer numbers are zero-based. Profiles are separate from the throughput measurements.
The focused C2 CUDA-Graph microbenchmark measures 2.51 us for BS1 × 6 verify rows on GB300; it is a kernel latency, not an end-to-end speedup.
Reproduce performance
Use identical checkpoint/dependencies for both checkouts and clear inherited
SGLANG_*overrides before setting the following:Use the pinned random input and client below. The client freezes GC after startup, flushes the request cache before every round, uses temperature 0 / ignore_eos / stream_interval 1, and checks the actual output length. Acceptance is simulated, not measured from real model agreement.
Prompt file SHA256:
2331eb918994e153dc659a4bd97f5eca87af7751268a5d931a1863a865801307.Input IDs SHA256 (compact JSON):
c37b49585df8fb22e125ae467003a00fff94a3f010cc7cddc03a3b463d047614.Restart the candidate server once and repeat into a separate results directory to reproduce the two-launch protocol.
Accuracy reproduction and evidence
The accuracy reproduction bundle records the server/client commands, generation parameters, dataset/prompt hashes, aggregate results and compact per-response records. Full responses and source/package manifests are retained locally for audit. The performance bundle contains all retained per-round timings and the exact random input IDs.
CI status
Changed-file pre-commit checks passed. The current GitHub GPU CI gate requires main commit
3700c4ee26a1, which thedsv4.1base does not contain (gate log). Repository-wide lint reports pre-existing formatting indeepseekv41_detector.py,test_serving_chat.pyandtest_deepseekv41_detector.py, none changed by this PR (lint log). The GPU CI jobs did not run; the GPU tests and model evaluations above were run directly on the assigned GB300 node.CI States
Latest PR Test (Base): ❌ Run #34595357264
Latest PR Test (Extra): ❌ Run #34595356901
Latest PR Test (AMD ROCm 10): ❌ Run #34595357091